Evaluation control

ABSTRACT

Systems and methods for evaluation control are disclosed herein. An evaluation control system can be used in evaluating one or several artifacts. The evaluation control system can be used to verify an evaluation and/or to provide evaluation training. The evaluation system can include a processor and a user device. The evaluation system can provide one or several artifacts and evaluation criteria to an evaluator. The evaluation system can receive one or several tags associated with the one or several artifacts and/or the evaluation criteria. The one or several tags can include information that, in connection with the evaluation criteria, support an evaluation and/or score for the one or several artifacts.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/811,347, filed on Apr. 12, 2013, and entitled “EVALUATION CONTROL,”the entirety of which is hereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

This disclosure relates in general to learning and can includetraditional classroom learning or on-line or computerized learningincluding, but without limitation, learning or instruction with aLearning Management System (LMS) and/or Online Homework System (OHS).

Work product is frequently generated during the learning process. Theevaluation of the work product is important to facilitating learning andproviding feedback to the creator of the work product. While evaluationof work product is important to the learning process, it also requiressignificant resources. Thus, better systems, methods, and devices aredesired to facilitate in evaluation of work product.

BRIEF SUMMARY OF THE INVENTION

In some aspects, the present disclosure relates to a system forverifying the evaluation of subject matter. The system includes aprocessor that receives a portfolio that includes a compilation ofartifacts that are work product of an evaluatee, provides one of theartifacts to an evaluator, and receives a plurality of first indicationsof an evaluation criteria, which evaluation criteria can include aplurality of sub-criteria. In some embodiments, the first indications ofthe evaluation criteria identify a first portion of the artifact and afirst portion of the evaluation criteria. The processor can furtherassign a value to the plurality of indications of the evaluationcriteria, which value identifies the source of the indications of theevaluation criteria. In some embodiments, the source of the indicationsof the evaluation criteria is the evaluator. The processor can receive afirst evaluation of the artifact, which evaluation can be based on theevaluation criteria and the first indications of the evaluationcriteria; receive a second evaluation of the artifact, secondindications of the evaluation criteria, and second associated portionsof the artifact; and compare the first and second evaluations of theartifact, the first and second indications of the evaluation criteria,and the first and second associated portions of the artifact. Theprocessor can provide an indication of the differences between the firstand second evaluations of the artifact, between the first and secondindications of the evaluation criteria, and between the first and secondthe associated portions of the artifact. The system can include a memorythat can store information relating to the received portfolio, the firstevaluation of the artifact, the second evaluation of the artifact, thesecond indications of the evaluation criteria and the second associatedportions of the artifact, and the result of the comparison of the firstand second evaluations of the artifact.

In some embodiments of the system, the processor can select theevaluator, which evaluator can be selected from one or several list ofpotential evaluators according to one or several traits of the evaluatorand/or of the artifacts being evaluated. In some embodiments of thesystem, the identified portion of the evaluation criteria can be asub-criterion.

In some embodiments, the system can include a user device that candisplay the first evaluation of the artifact, display the firstindications of the evaluation criteria, and display first associatedportions of the artifact. In some embodiments, the comparison of thefirst and second evaluations of the artifact can include generation of adifference value that characterizes the degree of difference between thefirst and second evaluations, retrieval of an acceptance threshold thatis a value demarking levels of acceptable and unacceptable differencesbetween the first and second evaluations, and comparison of thedifference value to acceptance threshold. In some embodiments, thesystem can include comparing the first indications of the evaluationcriteria and first associated portions of the artifact with the secondindications of the evaluation criteria and second associated portions ofthe artifact if the comparison of the difference value to the acceptancethreshold indicates an unacceptable level of difference between thefirst and second evaluations. In some embodiments, the processor cangenerate a final score based on the received plurality of firstindications of an evaluation criteria.

Some aspects of the present disclosure relate to a method of verifyingthe evaluation of subject matter. The method includes receiving aportfolio that includes a compilation of artifacts that are work productof an evaluatee, providing one of the artifacts to an evaluator, andreceiving a plurality of first indications of an evaluation criteria. Insome embodiments, the evaluation criteria can be a plurality ofsub-criteria, and the first indications of the evaluation criteria canidentify a first portion of the artifact and a first portion of theevaluation criteria. In some embodiments, the method can includeassigning a value to the plurality of indications of the evaluationcriteria, which value can identify the source of the indications of theevaluation criteria; receiving a first evaluation of the artifact, whichevaluation is based on the evaluation criteria and the first indicationsof the evaluation criteria; and providing the first evaluation of theartifact, the first indications of the evaluation criteria, and firstassociated portions of the artifact. In some embodiments, the method caninclude receiving a second evaluation of the artifact, secondindications of the evaluation criteria, and second associated portionsof the artifact; comparing the first and second evaluations of theartifact, indications of the evaluation criteria, and the associatedportions of the artifact; and providing an indication of the differencesbetween the first and second evaluations of the artifact, indications ofthe evaluation criteria, and the associated portions of the artifact.

In some embodiments, the method can include selecting the evaluator. Insome embodiments of the method, the identified portion of the evaluationcriteria can be a sub-criterion. In some embodiments of the method, thecomparison of the first and second evaluations of the artifact caninclude generating a difference value that characterizes the degree ofdifference between the first and second evaluations, retrieving anacceptance threshold that demarks levels of acceptable and unacceptabledifferences between the first and second evaluations, and comparing thedifference value to the acceptance threshold. In some embodiments, themethod includes comparing the first indications of the evaluationcriteria and first associated portions of the artifact with the secondindications of the evaluation criteria and second associated portions ofthe artifact if the comparison of the difference value to the acceptancethreshold indicates an unacceptable level of difference between thefirst and second evaluations. In some embodiments, the method includesgenerating a final score based on the received plurality of firstindications of an evaluation criteria.

Some aspects of the present disclosure relate to a method of training anevaluator. The method includes generating a portfolio including acompilation of artifacts that are work product of an evaluate; receivingevaluation criteria associated with the portfolio, which evaluationcriteria can include a plurality of sub-criteria; and receiving a keythat includes a plurality of indications of an evaluation criteria. Insome embodiments, the indications in the key are the correct indicationsof an evaluation criteria for a portfolio. In some embodiments, theindications of the evaluation criteria in the key identify a portion ofone of the artifacts and a sub-criterion of the evaluation criteria.Some embodiments of the method include providing the portfolio to atrainee and receiving a portfolio evaluation that includes a pluralityof indications of the evaluation criteria. In some embodiments, theindications in the portfolio evaluation are received from the trainee,and the indications of the evaluation criteria in the portfolioevaluation identify a portion of one of the artifacts and asub-criterion of the evaluation criteria. Some embodiments of the methodinclude comparing the key and the portfolio evaluation according to aBoolean function to determine the accuracy of the portfolio evaluationand providing an indicator of the accuracy of the portfolio evaluation.

In some embodiments of the method, the comparison of the key and theportfolio evaluation includes comparing the score of the artifact in theportfolio evaluation with the score of the artifact in the key. In someembodiments, the comparison of the portfolio evaluation and the keyincludes generating a difference value that characterizes the degree ofdifference between the portfolio evaluation and the key, retrieving anacceptance threshold that demarks levels of acceptable and unacceptabledifferences between the portfolio evaluation and the key, and comparingthe difference value to an acceptance threshold. In some embodiments,the comparison of the key and the portfolio evaluation includes thecomparison of the indications of the evaluation criteria in theportfolio evaluation to the indications of the evaluation criteria inthe key. In some embodiments, the method includes providing additionaltraining material.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 is a block diagram illustrating one embodiment of an evaluationcontrol system.

FIG. 2 is a block diagram illustrating one embodiment of a user devicefor use with an evaluation control system.

FIG. 3 is a flowchart illustrating one embodiment of a process forevaluation of one or several portfolios and/or artifacts.

FIG. 4 is a flowchart illustrating one embodiment of a process forevaluation control.

FIG. 5 is a flowchart illustrating one embodiment of a process forgenerating evaluation data.

FIG. 6 is a flowchart illustrating one embodiment of a process fordisplaying evaluation data.

FIG. 7 is a flowchart illustrating one embodiment of a process fortraining an evaluator

FIG. 8 is a block diagram illustrating one embodiment of a computersystem.

FIG. 9 is a block diagram illustrating one embodiment of aspecial-purpose computer.

In the appended figures, similar components and/or features may have thesame reference label. Where the reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same reference label. Further, various componentsof the same type may be distinguished by following the reference labelby a dash and a second label that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION OF THE INVENTION

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

In one embodiment, the present disclosure provides a method forverifying the evaluation of the portfolio. This method can includereceiving a portfolio that includes work product which can be, forexample, generated by one or several individuals. In some embodiments,all or portions of this work product is provided to an evaluator and oneor several tags are received from the evaluator, which tags identifysubject matter relevant to an evaluation criteria. In some embodiments,the evaluation of the portfolio and/or of one or several of theartifacts is received and used to generate first evaluation data. Thisfirst evaluation data can be compared to second evaluation data whichcan be, for example, generated by a second evaluator. The degree ofdifferences and the differences between the first evaluation data andthe second evaluation data can, in some embodiments, provide informationrelating to the accuracy evaluation. Specifically, the evaluation ismore likely accurate if the first and second evaluation data matchand/or closely match.

In one embodiment, the present disclosure provides a method for trainingan evaluator, also referred to herein as a trainee, to evaluate aportfolio. This method can include generating a training portfolio,evaluation criteria associated with the portfolio, and a second,verified evaluation, also referred to herein as a key. In someembodiments, this method can include receiving an indication of tagsidentifying a portion of the portfolio, a relevant portion of theevaluation criteria, and the trainee, and receiving a portfolioevaluation. The received portfolio evaluation can be compared to the keyand differences between the received evaluation and the key can beindicative of the training level of the trainee and/or potential areasfor providing further training to the trainee. In some embodiments,these differences, and the degree of difference can be used to determinewhen a trainee is a trained evaluator.

With reference now to FIG. 1, a block diagram of one embodiment of anevaluation control system 100 is shown. The evaluation control system100 collects, receives, and stores data relating to an artifact. In someembodiments, the artifact can comprise any work product and can include,for example, digital work product. In some embodiments, the digital workproduct can include written work product and/or recorded work productwhich can include sound and/or video recordings. In some embodiments,the evaluation control system 100 can collect, receive, and store datarelating to and/or facilitating the evaluation of one or severalartifacts. The evaluation control system 100 can receive and/or createevaluation criteria for one or several artifacts, can receive one orseveral indicators of portions of the artifact relevant to theevaluation criteria, and can receive an evaluation for one or severalartifacts and/or portfolios. In some embodiments, the evaluation controlsystem 100 can be configured to assess the evaluation of the one orseveral artifacts to facilitate training of one or several evaluatorsand/or for evaluation quality control.

The evaluation control system 100 can include a processor 102. Theprocessor 102 can provide instructions to and receive information fromthe other components of the evaluation control system 100. The processor102 can act according to stored instructions, which stored instructionscan be located in memory associated with the processor and/or in othercomponents of the evaluation control system 100. The processor 102 cancomprise a microprocessor, such as a microprocessor from Intel® orAdvanced Micro Devices, Inc.®, or the like.

The evaluation control system 100 can include one or several databases104. The one or several databases 104 can comprise stored data relevantto the functions of the evaluation control system 100. The one orseveral databases 104 include a profile database 104-A. The profiledatabase 104-A can include profile data for one or several users whichcan include, for example, one or several evaluators and/or one orseveral supervisors. In some embodiments, for example, the profiledatabase 104-A can include information relating to the creator of thework product, also referred to herein as a evaluatee.

The profile data can include any information relating to the user; insome embodiments, for example, this information can include anindication of the evaluator's progress through a training program, anindication of the quality and/or accuracy of one or several of theevaluator's evaluations, and/or an indication of the type and/or subjectmatter of artifacts that the evaluator may evaluate.

In some embodiments, for example, the profile database 104-A cancomprise login information. This information can include, for example,information identifying a user such as, for example, a username andpassword or a user identification number. In some embodiments, forexample, when a user desires to access the evaluation control system100, the user can be prompted to enter identification information suchas, for example, a username and password. After the user provides theidentification information, the evaluation control system 100 can verifythe identification information, and specifically, the processor 102 cancompare the user-provided identification information to informationstored within the profile database 104-A to determine if the actual useris an authorized user.

The one or several databases 104 can include a portfolio database 104-B.The portfolio database 104-B can include one or several portfolios. Insome embodiments, a portfolio can comprise a grouping of one or severalartifacts. In some embodiments, these artifacts can be grouped accordingto type, content, evaluation type which can include information relatingto how the artifacts in the portfolio can be evaluated, and/or workproduct author.

The evaluation control system 100 can include a tag database 104-C. Thetag database 104-C can include information used in evaluating one orseveral artifacts. In some embodiments, for example, the tag database104-C can comprise a plurality of tags. In some embodiments, the tagscan be indications of the application of one or several evaluationcriteria or sub-criteria to all or portions of an artifact. In someembodiments, a tag can identify a portion of an artifact, the tag canidentify a portion of an evaluation criterion relevant to the indicatedportion of the work product, and in some embodiments, the tag canindicate the evaluator who created the tagging, the time and date of thetag creation, and/or any other desired information relating to the tag.In some embodiments, the tag can include information such as, forexample, one or several comments, notes, and/or marks. These comments,notes, and/or marks can be created by the user and can, for example,provide feedback to the creator of the artifact and/or be used inscoring/evaluation of the artifact. In some embodiments, the tag caninclude a tag type. In some embodiments, the tag type can associate thetag with, for example, a portion of a rubric, a positive or negativeattribute, or the like. In one embodiment, for example, the tag type canindicate a misspelling; a grammatical error; a content error; a good,poor, or adequate technique; a good, average, or bad argument; a good,average, or poor use of content; a correct answer; an incorrect answer;or the like.

The evaluation control system 100 can include an evaluation database104-D. The evaluation database 104-D can include information that canfacilitate the performing of an evaluation of one or several artifactsand/or portfolios. In some embodiments, this information can include oneor several evaluation criteria for use in evaluating one or severalartifacts and/or portfolios. In some embodiments, the evaluationcriteria can comprise, for example, a rubric and/or other scoring aid.The evaluation criteria can, in some embodiments, comprise one orseveral sub-criteria that can, for example, focus on a specific aspectof the evaluation.

The evaluation database 104-D can include information identifying theresult of a started, partially completed, and/or completed evaluation.In some embodiments, the information in the evaluation database 104-Dcan be organized by portfolio, artifact, evaluator, or in any otherdesired manner. The evaluation database 104-D can include informationlinking one or several artifacts' and/or portfolios' tags with one orseveral evaluation criteria and/or the result of the evaluation of oneor several artifacts and/or portfolios.

The evaluation database 104-D can include information used in assessingan evaluation of one or several artifacts and/or portfolios. In someembodiments, for example, this information can include a secondevaluation of the one or several artifacts and/or portfolios. In someembodiments, this second evaluation can be a verified second evaluationthat reflects a desired and/or ideal evaluation, and in someembodiments, the second evaluation can be a non-verified evaluation.

The evaluation control system 100 can include one or several userdevices 106, which can include an evaluator device 106-A and/or asupervisor device 106-B. The user devices 106 allow a user, including anevaluator, a supervisor, a trainer, and/or a trainee to access theevaluation control system 100. The details and function of the userdevices 106 will be discussed at greater length in reference to FIG. 2below.

The evaluation and optimization system 100 can include a data source108, also referred to as a repository. The data source 108 can be thesource of the one or several artifacts and/or portfolios, the source ofthe one or several evaluation criteria, and/or the source of one orseveral second evaluations. In some embodiments, the data source cancomprise an educational service provider, such as, for example, aschool, a university, a college, and/or a Learning Management System(LMS).

The evaluation control system 100 can include a network 110. The network110 allows communication between the components of the evaluationcontrol system 100. The network 110 can be, for example, a local areanetwork (LAN), a wide area network (WAN), a wired network, wirelessnetwork, a telephone network such as, for example, a cellphone network,the Internet, the World Wide Web, or any other desired network. In someembodiments, the network 110 can use any desired communication and/ornetwork protocols.

With reference now to FIG. 2, a block diagram of one embodiment of auser device 106 is shown. As discussed above, the user device 106 can beconfigured to provide information to and/or receive information fromother components of the evaluation control system 100. The user devicecan access the evaluation control system 100 through any desired meansor technology, including, for example, a webpage, a web portal, or vianetwork 110. As depicted in FIG. 2, the user device 106 can include anetwork interface 200. The network interface 200 allows the user device106 to access the other components of the evaluation control system 100,and specifically allows the user device 106 to access the network 110 ofthe evaluation control system 100. The network interface 200 can includefeatures configured to send and receive information, including, forexample, an antenna, a modem, a transmitter, receiver, or any otherfeature that can send and receive information. The network interface 120can communicate via telephone, cable, fiber-optic, or any other wiredcommunication network. In some embodiments, the network interface 200can communicate via cellular networks, WLAN networks, or any otherwireless network.

The user device 106 can include a user interface 202 that communicatesinformation to, and receives inputs from, a user. The user interface 202can include a screen, a speaker, a monitor, a keyboard, a microphone, amouse, a touchpad, a keypad, or any other feature or features that canreceive inputs from a user and provide information to a user.

The user device 106 can include a review engine 204. In someembodiments, the review engine 204 can be configured to receive one orseveral artifacts and/or portfolios from the portfolio database 104-Band provide the one or several artifacts and/or portfolios to the uservia, for example, the user interface 202. In some embodiments, thereview engine can include features and/or software that allow providinga range of software and/or artifact types to the user including, forexample, images, written documents, recordings including sound and/orvideo recording, and/or any other desired software and/or artifact type.

The user device 106 can include a tagging engine 206. In someembodiments, the tagging engine can be configured to allow a user toadd, remove, and/or edit a tag that can be, for example, associated withone or several artifacts and/or one or several portfolios. In someembodiments, for example, the tagging engine 206 can be configured toallow a user to tag a portion of one or several artifacts and/orportfolios. In some embodiments the tag can identify a portion of theone or several artifacts and/or portfolios including, for example, astarting point, an ending point, and/or a duration of the portion of theone or several artifacts and/or portfolios. In some embodiments, thetagging engine 206 can be configured to allow a user to associate theportion of one or several artifacts and/or portfolios with one orseveral evaluation criteria and/or evaluation sub-criteria.

The user device 106 can include an evaluation engine 208. The evaluationengine 208 can be configured to allow a user to evaluate one or severalartifacts and/or portfolios. In some embodiments, for example, theevaluation engine can be configured to group one or several tagsassociated with one or several artifacts and/or portfolios and providethese grouped tags to the user. In one embodiment, for example, theevaluation engine 208 can group tags associated with one artifact andwith one evaluation criteria and/or evaluation sub-criteria, and providethese tags to the user. The evaluation engine 208 can be configured toallow the user to review the portions of the one or several artifactsand/or portfolios associated with the tags, and to receive an evaluationfrom the user based on those tags.

In some embodiments, the evaluation engine 208 can be configured toallow the review of an evaluation of one or several artifacts and/orportfolios. In some embodiments, for example, the evaluation engine 208can be configured to compare the evaluation of one or several artifactsand/or portfolios with a second evaluation that can be, for example, theverified evaluation. In some embodiments, the evaluation engine 208 canbe further configured to generate and provide a comparison reportidentifying the differences between the evaluation and the secondevaluation and indicating whether the evaluation is acceptable.

With reference now to FIG. 3, a flowchart illustrating one embodiment ofa process 300 for evaluation of one or several portfolios and/orartifacts is provided. In some embodiments, the process 300 can beperformed by the evaluation control system 100 and/or components of theevaluation control system 100.

The process 300 begins at block 302 wherein a portfolio is received. Insome embodiments, for example, the portfolio can be received by and/orfrom a component of the evaluation control system 100, and in oneembodiment, the portfolio can be received from the data source 108. Insome embodiments, and as discussed above, the portfolio can comprise oneor several artifacts which can be a collection of work product. In someembodiments, this work product can be generated by a user of the datasource 108, and in some embodiments, this work product can be collectedby the data source 108. In some embodiments, after the portfolio hasbeen received, the portfolio can be stored within one of the databases104 including, for example, the portfolio database 104-B.

After the portfolio has been received, the process 300 proceeds to block304 wherein the artifacts are provided. In some embodiments, forexample, the artifacts can be provided to the user via one of the userdevices 106 including, for example, the evaluator device 106-A. In someembodiments, the user device 106 can provide the artifacts to the uservia the user interface 202. The artifacts can be retrieved from the oneor several portfolios stored within the portfolio database 104-B. Insome embodiments, for example, the processor 102 can query the portfoliodatabase 104-B for a stored artifact. In some embodiments, one orseveral artifacts can be selected from the portfolio database 104-B andcan be provided to the user.

After the artifacts have been provided, the process 300 proceeds toblock 306 wherein a tag is received and/or applied. In some embodiments,for example, the tag can be received via one of the user devices 106such as, for example, the evaluator device 106-A, and can be stored inone or several of the databases 104 including tag database 104-C. In oneembodiment, a tag can be applied in that the tag, and the data relevantto the tag, is stored in one of the databases 104. In one embodiment,for example, and as discussed above, the tag can identify a portion ofthe artifact, can identify a portion of the evaluation criteria relevantto the portion of the artifact, can include a note relating to theevaluation criteria and/or to the tagged portion of the artifact, and/orcan identify the user adding, removing, and/or editing the tag.

After the tag has been received, the process 300 proceeds to block 308wherein the evaluation is applied. In some embodiments, for example, theevaluation can be applied based on the tags associated with the artifactand/or stored in the tag database 104-C. In some embodiments, theevaluation can be applied based on the number of tags associated withone or several of the evaluation criteria and/or sub-criteria and/orbased on information relating to the evaluation criteria and/orsub-criteria that do not have a related tag and/or have fewer relatedtags than a threshold value. The application of the evaluation can, insome embodiments, be received from the user via the user device 106and/or generated by the processor 102.

With reference now to FIG. 4, a process 400 for evaluation control isprovided. In some embodiments, for example, the process 400 forevaluation control can be performed by the evaluation control system 100and/or a component of thereof. The process 400 begins at block 402wherein the portfolio is received. In some embodiments, for example, theportfolio can be received by and/or from a component of the evaluationcontrol system 100, and in one embodiment, the portfolio can be receivedfrom the data source 108. In some embodiments, and as discussed above,the portfolio can comprise one or several artifacts which can be acollection of work product. In some embodiments, this work product canbe generated by a user of the data source 108, and in some embodiments,this work product can collected by the data source 108. In someembodiments, after the portfolio has been received, the portfolio can bestored within one of the databases 104 including, for example, theportfolio database 104-B.

After the portfolio has been received, the process 400 proceeds to block404 wherein an indication of artifacts is provided. In some embodiments,for example, the indication of artifacts can be provided to the user viaone of the user devices 106 such as, for example, via the evaluatordevice 106-A. The indication of artifacts can comprise an indicator ofartifacts stored within the portfolio database 104-B. In someembodiments, this indicator can comprise a listing, table, and/or indexof artifacts stored in the portfolio database 104-B. The indication ofthe artifacts can be retrieved from the one or several portfolios storedwithin the portfolio database 104-B. In some embodiments, for example,the processor 102 can query the portfolio database 104-B for indicationsof artifacts stored within the portfolio database 104-B. Theseindications of artifacts stored within the portfolio database 104-B canbe provided to the user via, for example, the user interface 202.

After the indication of the artifacts has been provided, the process 400proceeds to block 406 wherein a selection of one or several of theartifacts is received. In some embodiments, for example, the selectionof one or several of the artifacts can be received via one of the userdevices 106 such as, for example, the evaluator device 106-A. Theselected one or several artifacts can correspond to provided indicationsof artifacts stored within the portfolio database.

After selection of one or several artifacts has been received, theprocess 400 proceeds to block 408 wherein an artifact is provided. Insome embodiments, for example, the artifacts can be provided to the uservia one of the user devices 106 including, for example, the evaluatordevice 106-A. In some embodiments, the user device 106 can provide theartifacts to the user via the user interface 202. The artifacts can beretrieved from the one or several portfolios stored within the portfoliodatabase 104-B. In some embodiments, for example, the processor 102 canquery the portfolio database 104-B for a stored artifact. In someembodiments, one or several artifacts can be selected from the portfoliodatabase 104-B and can be provided to the user.

After the artifacts have been provided, the process 400 proceeds toblock 410 wherein a tag is received and/or applied. In some embodiments,for example, the tag can be received from the user via the user device106, and specifically via, for example, the evaluator device 106-A. Insome embodiments, the tag can identify a portion of one or severalartifacts and/or portfolios, a portion of a relevant evaluation criteriaand/or evaluation sub-criteria, and/or an indicator of theidentification of the person and/or evaluator adding, removing, and/orediting the tag. In some embodiments, the tag can be applied in that thetag and/or the portion of the artifact associated with the tag isstored. In some embodiments, the received tag can be stored in one ofthe databases 104 including, for example, the tag database 104-C.

After the tag has been received, the process 400 proceeds to block 412wherein the tag is correlated to the evaluation criteria. In someembodiments, for example, this correlation can include retrieving taginformation identifying a related one or several evaluation criteriaand/or evaluation sub-criteria and storing this information within theevaluation database. In some embodiments, for example, this can beperformed by the processor 102 and/or by another component of theevaluation control system 100 including, for example, by the user device106 and/or component thereof such as the tagging engine 206 and/or theevaluation engine 208.

After the tag has been correlated to the evaluation criteria, theprocess 400 proceeds to block 414 wherein an evaluation of the artifactand/or portfolio is received. In some embodiments, for example, thisevaluation can be received from and/or performed with one of the userdevices 106 and/or other components of the evaluation control system100. In some embodiments, for example, this step can include thegrouping of one or several tags, providing the group of one or severaltags to the user, and receiving an evaluation based on these tags andthe evaluation criteria. In some embodiments, the receipt evaluation canbe stored in one of the databases 104 such as, for example, theevaluation database 104-D.

After the evaluation of the artifact and/or portfolio has been received,the process 400 proceeds to block 416 wherein portions of the one orseveral artifacts and/or portfolios identified by the one or severaltags are saved. In some embodiments, for example, these portions of theartifact can be saved within one of the databases 104, and specificallywithin the portfolio database 104-B and/or the tag database 104-C.

After portions of the artifact identified by one or several tags havebeen saved, the process 400 proceeds to block 418 wherein a firstevaluation data is generated. In some embodiments, for example, thefirst evaluation data can comprise information relating to theevaluation and allowing the re-creation of the evaluation. Thisinformation can include the evaluation of one or several artifactsand/or portfolios provided by the evaluator, the one or several tagsassociated with the one or several artifacts and/or portfoliosidentified by the evaluator, saved portions of the one or severalartifacts and/or portfolios identified by the one or several tags,and/or an indicator of the identity of the evaluator.

After the first evaluation data has been generated, the process 400proceeds to decision state 420 wherein it is determined if secondevaluation data has been received. In some embodiments, for example,this determination can be made by the processor 102 and can includequerying one or several of the databases 104 including, for example, theevaluation database 104-B.

If it is determined that there is no second evaluation data, the process400 proceeds to block 422 wherein evaluation data is provided. In someembodiments, for example, the evaluation data can be provided to one orseveral of the user devices 106 including, for example, the evaluatordevice 106-A and/or the supervisor device 106-B.

Returning again to decision state 420 it is determined that secondevaluation data has been received, the process 400 proceeds to block 424wherein the evaluation data are compared. In some embodiments, forexample, this comparison can include comparing the evaluation of the oneor several artifacts and/or portfolios to determine if the evaluation isthe same, the comparison of the tags associated with the evaluationcriteria of the first and second evaluation data to determinesimilarities and/or differences in tags applied in both instances,and/or the comparison of portions of the one or several artifacts and/orportfolios identified by the tags.

In some embodiments, for example, this comparison of the evaluation datacan include comparison of the tags associated with the first and secondevaluation data to determine the similarities/differences in the twotaggings of the artifact. In one embodiment, for example, thiscomparison can include determining whether the tags associated with thetwo evaluations each identify the same positive and/or negativeaspects/attributes of the artifact. In some embodiments, this comparisoncan include comparing the linking between the content of the artifact tothe evaluation criteria to determine whether the evaluators linkedsimilarly tagged content to similar portions of the evaluation criteria.Similarly, in some embodiments, this comparison can include a comparisonof the overall evaluation and/or score for the artifact and/orportfolio. In one embodiment, for example, the overall evaluationsand/or scores for the artifact and/or portfolio can be compared. In someembodiments, if the comparison of the overall evaluations and/or scoresfor the artifact and/or portfolio indicate sufficient difference, thenthe association of the tags with content of the artifact and/orportfolio and/or the association of the tags with the evaluationcriteria are compared.

In one such embodiment, the overall evaluation and/or score of the firstand second evaluations are compared to generate a difference valueindicating the degree of difference between the overall evaluationand/or score of the first and second evaluations. In some embodiments,this difference value is compared to an acceptance threshold. In someembodiments, the acceptance threshold can identify a degree ofdifference between evaluations that identifies acceptable/unacceptabledifferences between the evaluations and/or triggers additionalcomparison of the first and second evaluations. In one embodiment, ifthe comparison of the difference value to the acceptance thresholdindicates that the evaluations are adequately similar, then the processproceeds to decision state 426, discussed at greater length below,whereas, if the comparison of the difference value to the acceptancethreshold indicates that the evaluations are inadequately similar, thenthe comparison of the evaluation data can include a comparison of thetags as discussed above.

In some embodiments, for example, this comparison of the evaluation datacan be performed by the processor 102 and/or by the evaluation engine208 of one of the user devices 106. In some embodiments, for example,this comparison can be performed according to a Boolean function whereinmatching aspects of the first and second evaluation data are assigned afirst value and nonmatching aspects of the first and second evaluationdata are assigned a second value.

After the first and second evaluation data have been compared, theprocess 400 proceeds to decision state 426 wherein is determined ifthere are differences between the first and second evaluation data. Insome embodiments, for example, this determination can includedetermining whether there are any differences between the first andsecond evaluation data which can include, for example, determiningwhether the evaluation of the one or several artifacts and/or portfoliosare different, determining whether one or several applied tags aredifferent, and/or determining whether the portions of the one or severalartifacts and/or portfolios identified by the tags are different. Insome embodiments, for example, this determination can be performed withreference to values assigned to the first and second evaluation data asdiscussed above, and can be performed by the processor 102 and/or theevaluation engine 208. If it is determined that there are no differencesand/or that a difference threshold has not been met, then the process400 proceeds to block 422 when evaluation data is provided.

If it is determined that there are differences between the first andsecond evaluation data and/or that the difference threshold has beenmet, then the process 400 proceeds to block 428 wherein a differencereport is generated. In some embodiments, the difference report can, forexample, identify the differences between the first and secondevaluation data including, for example, differences in the evaluation ofthe one or several artifacts and/or portfolios, differences in the oneor several artifacts and/or portfolios, and/or differences in theportions of the artifacts and/or portfolios tags identified by the tags.In some embodiments, the difference report can be generated by theevaluation control system 100, and can specifically be generated by theprocessor 102 and/or one of the user devices 106 or component thereofincluding, for example, the evaluation engine 208.

After the difference report has been generated, the process 400 proceedsblock 430 wherein the difference report is provided. In someembodiments, for example, the difference report can be provided to theuser via one of the user devices including, for example, the supervisordevice 106-B, and specifically the user interface 202 of the user device106.

After the difference report has been provided, the process 400 proceedsto decision state 432 wherein it is determined if additional evaluationdata has been received. In some embodiments, the additional evaluationdata can be, for example, third evaluation data, fourth evaluation data,fifth evaluation data, sixth evaluation data, and/or any otherevaluation data including, for example, n^(th)-evaluation data. In someembodiments, for example, evaluation data can be collected until thereis a convergence of the evaluation data.

In some embodiments, additional evaluation data can be data relating toa further evaluation of one or several artifacts and/or portfolios thatcan be used to determine which of the first and/or second evaluationdata is accurate, accurately reflects a correct evaluation of one orseveral artifacts and/or portfolios, and/or is the most accurate. Insome embodiments, for example, this determination can be made by theprocessor 102 and can include querying one or several of the databases104 including, for example, the evaluation database 104-B. If it isdetermined that there is no additional evaluation data, the process 400can terminate.

If it is determined that there is additional data, the process 400proceeds to block 434 wherein the evaluation dated is compared. In someembodiments, for example, this comparison can include comparing one orboth of the first and second evaluation data with the additionalevaluation data. In some embodiments, for example, this comparison caninclude comparing the evaluation of the one or several artifacts and/orportfolios to determine if the evaluation is the same, comparing thetags associated with the evaluation criteria of the first, second,and/or additional evaluation data to determine similarities and/ordifferences in tags applied in those instances, and/or the comparison ofportions of the one or several artifacts and/or portfolios identified bythe tags. In some embodiments, for example, this comparison can beperformed by the processor 102 and/or by the evaluation engine 208 ofone of the user devices 106. In some embodiments, for example, thiscomparison can be performed according to a Boolean function whereinmatching aspects of the first, second, and/or additional evaluation dataare assigned a first value and nonmatching aspects of the first andsecond evaluation data are assigned a second value.

After the evaluation data has been compared, the process 400 can proceedto block 436 wherein a discrepancy report is provided. In someembodiments, for example, the discrepancy report can identifydifferences between the first and second evaluation data and theadditional evaluation data. In some embodiments, for example, thediscrepancy report can identify which of the first and/or secondevaluation data most closely approximates the additional evaluationdata. In some embodiments, for example, the discrepancy report can beprovided to the user via one of the user devices 106 including, forexample, the supervisor device 106-B, and specifically the userinterface 202 of the user device 106.

With reference now to FIG. 5, a flowchart illustrating one embodiment ofa process 500 for generating evaluation data is shown. In someembodiments, the process 500 can be performed as part of, or in theplace of block 418 shown in FIG. 4. The process 500 can be performed bythe processor 102, the evaluation engine 208 of one of the user devices106, and/or by any other component of the evaluation control system.

The process 500 begins at decision state 502, wherein it is determinedwhether to display data, and particularly, whether to display all orportions of the evaluation data including, for example, the firstevaluation data. In some embodiments, this can include determiningwhether a user request for the display of evaluation data has been made,which user request can be, a specific user request, or a general rule orrequest to display evaluation data. If it is determined that evaluationdata will not be displayed, then the process 500 proceeds to block 504and returns to block 420 or FIG. 4.

If it is determined that evaluation data will be displayed, then theprocess 500 proceeds to block 506, wherein artifacts are received. Insome embodiments, this can include receiving all artifacts for which theevaluation data is relevant including, for example, all evaluatedartifacts for, a student, a class, a grade, a study, or any other groupof artifacts. In some embodiments, these artifacts can be retrievedfrom, for example, one of the databases 104 such as, for example, theportfolio database 104-B.

After the artifacts have been received/retrieved, the process 500proceeds to block 508, wherein the tags are received/retrieved. In someembodiments, the retrieval of the tags can include the retrieval ofinformation associated with the tags, including, for example, one orseveral comments, notes, or marks created and/or associated with thetags. In some embodiments, the tags can be retrieved from one of thedatabases such as, for example, the tag database 104-C.

After the tags have been received, the process 500 proceeds to block 510where grouping criteria are received. In some embodiments, the groupingcriteria can include one or several rules for categorizing tags. In someembodiments, these one or several rules can categorize tags according tothe artifact with which a tag is associated, the type of tag, the typeof comment, note, or mark associated with the tag, or the like. Thegrouping criteria can be created by the user and can be received via oneor several of the user devices 106 and can be stored in one of thedatabases 104 such as the evaluation database 104-D.

After the grouping criteria have been received, the process 500 proceedsto block 512 wherein the tags are grouped. In some embodiments, thegrouping of the tags can include, for example, grouping the tagsaccording to one or several attributes of the tag including, forexample, the tag type, tag content, including any comment, note, or markassociated with the tag, or the like. In some embodiments, this groupingcan be performed according to the grouping criteria. In someembodiments, the grouping of the tags can include storing informationidentifying the grouping of one or several of the tags, which data canbe stored in, for example, the tag database 104-C.

After the tags have been grouped, the process 500 proceeds to block 514,wherein the tags are displayed. In some embodiments, the tags can bedisplayed to the user via, for example, the user device 106. In oneembodiment, for example, the tags and artifacts can be simultaneouslydisplayed to the user such that the tags are located in a first displayportion and one or several artifacts, or portions thereof, are locatedin a second display portion. In some embodiments, the first and seconddisplay portions can be first and second portions of a display such as,for example, a screen or monitor. After the tags have been displayed,the process 500 proceeds to block 516 and returns to block 420 of FIG.4.

With reference now to FIG. 6, a flowchart illustrating one embodiment ofa process 600 for displaying evaluation data is shown. In someembodiments, the process 600 can be performed as part of, or in theplace of block 514 shown in FIG. 5. The process 600 can be performed bythe processor 102, the evaluation engine 208 of one of the user devices106, and/or by any other component of the evaluation control system.

The process 600 begins at block 602 wherein tag data is retrieved. Insome embodiments, this retrieval of tag data can be the same as thereceipt of tags in block 508, and in some embodiments, this retrieval oftag data can include the retrieval of tag data in addition to thatretrieved in block 508. This tag data can be retrieved from one of thedatabases 104 such as, for example, the tag database 104-C.

After the tag data has been received, the process 600 proceeds to block604 wherein a count is incremented for each tag associated withreceived/retrieved tag data. In some embodiments, this incrimination canbe performed based on all of the information received in one or both ofblocks 508 and 602. In some embodiments, the count can be stored in oneof the databases 104. After the count has been incremented for each ofthe received/retrieved tags, the process 600 proceeds to block 606wherein the number of tags is determined. In some embodiments, this canbe achieved via retrieval of the count.

After the number of tags has been determined, the process 600 proceedsto block 608 wherein tag type is extracted from the tag data. In someembodiments, this tag type information can be stored as part of the tagdata, and can be extracted from the tag data received in block 602. Insome embodiments, this extraction can be performed by the processor 102and/or the user device 106.

After the tag type has been extracted from the tag data, the process 600proceeds to block 610 wherein a tag score is generated. In someembodiments, the tag score can reflect the degree to which a tag affectsthe score of one or several artifacts. In some embodiments, the tagscore can indicate a degree to which the tag increments, decrements, ordoes not affect an artifact score. In some embodiments, for example, atag indicating a negative aspect of an artifact can, based on thestrength of the negative aspect artifact, decrease the score of theartifact. Similarly, a tag indicating a position aspect of an artifactcan, based on the strength of the positive aspect of the artifact,increase the score of the artifact. The tag score can be stored in oneof the databases 104 such as the tag database 104-C. In someembodiments, the tag score can be generated according to scoring rulesthat can be stored in one of the databases 104 such as, for example, theevaluation database 104-D.

After the tag score has been generated, the process 600 proceeds toblock 612, wherein a sum score is calculated. In some embodiments, thesum score can be a value representing the aggregate effect of some orall of the tags. The sum score can be a rough score that can beconverted to a final score for an artifact and/or a final score for anartifact. In some embodiments, the sum score can be calculated by thecombination of tag scores, which combination can include the addition oftag scores, subtraction of tag scores, and the application of one orseveral weighting factors to some or all of the tag scores based on therelative importance and/or weight associated with some or all of the tagscores. In some embodiments, the sum score can be stored in one of thedatabases 104 such as the tag database 104-C. The tag score can begenerated according to scoring rules that can be stored in one of thedatabases 104 such as, for example, the evaluation database 104-D.

After the sum score has been calculated, the process 600 proceeds toblock 614 wherein the sum score is compared to scoring data. In someembodiments in which the sum score is a rough score, this can includethe conversion of the sum score to a final score. The final score can,in some embodiments, be a recommended final score, and/or final scorerange. In some embodiments, an evaluator may be able to select a scoreother than the recommended final score, and in some embodiments, theevaluator may be limited to selecting a score corresponding to therecommended final score, including a score from the range indicated bythe recommended final score. In some embodiments, this conversion caninclude comparison of the sum score to scoring data, application of ascoring algorithm, or the like. In some embodiments, this conversion canbe performed by the processor 102 and/or user device 106.

After the sum score has been compared to the scoring data, the process600 proceeds to block 616, wherein the score is retrieved. In someembodiments, this score can be the final score, and the retrieval ofthis score can be the receipt of the result of the scoring algorithm,the output of the comparison of the sum score to the scoring data, orthe like. In some embodiments, the final score can be stored within oneof the databases 104 such as, for example, the evaluation database104-D. After the score has been retrieved, the process 600 proceeds toblock 618 and returns to block 420 of FIG. 4.

With reference now to FIG. 7, a flowchart illustrating one embodiment ofa process 700 for training an evaluator is shown. In some embodiments,this process 700 can be used to provide an evaluation task to a traineeand to qualify the results of that evaluation task. The process 700 canbe performed by the evaluation control system 100 and/or components ofthe evaluation control system 100. Process 700 begins at block 702wherein the portfolio is generated. In some embodiments, for example, aportfolio can be generated specifically for purposes of training anevaluator, which evaluator in training is also referred to herein as atrainee. In some embodiments, the generated portfolio can be createdwith the user device 106 such as, for example, the supervisor device106-B and/or with the data source 108.

After the portfolio has been generated the process 700 proceeds block704 wherein evaluation criteria are received. In some embodiments, theevaluation criteria can be used in evaluating one or several artifactsand/or portfolios and can comprise indications of features of the one orseveral artifacts and/or portfolios, and a scoring effect of thosefeatures. In some embodiments, the evaluation criteria can comprise, forexample, a rubric and/or other scoring aid. The evaluation criteria can,in some embodiments, comprise one or several sub-criteria that can, forexample, focus on a specific aspect of the evaluation. The evaluationcriteria can be created with the user device 106 such as, for example,the supervisor device 106-B and/or with the data source 108.

After the evaluation criteria has been received, the process proceeds toblock 706 wherein a key is received. In some embodiments, for example,the key can comprise a second evaluation of the generated portfolio, andspecifically, a verified evaluation of the portfolio. The key caninclude information relating to the evaluation the portfolio, andspecifically to the ideal overall evaluation of the portfolio and/or oneor several artifacts in the portfolio, information relating to idealtagging associated with the portfolio and the evaluation criteria, andportions of the portfolio and/or one or several artifacts indicated bythe ideal tags.

After the key has been received, the process 700 proceeds to block 708wherein the portfolio is provided. In some embodiments, for example, theportfolio and/or one or several artifacts in the portfolio can beprovided to the user via one of the user devices 106 including, forexample, the evaluator device 106-A. In some embodiments, the userdevice 106 can provide the portfolio and/or one or several artifacts inthe portfolio to the user via the user interface 202. The artifacts canbe retrieved from the one or several portfolio stored within theportfolio database 104-B. In some embodiments, for example, theprocessor 102 can query the portfolio database 104-B for the storedportfolio and/or one or several artifacts in the portfolio. In someembodiments, the desired portfolio and/or one or several artifacts inthe desired portfolio can be selected from the portfolio database 104-Band can be provided to the user.

After the portfolio has been provided, the process 700 proceeds toblocks 710 through 712 which blocks outline the step of receiving thetag indicated in block 410 of FIG. 4 in greater detail. The step ofreceiving the tag begins with block 710 wherein an indicator of one orseveral artifact and/or portfolio portions is received. In someembodiments, this indicator can be received via one of the user devices106 including, for example, the evaluator device 106-A. In someembodiments, this can include an indication of one or several taggedportions of the one or several artifacts and/or portfolios. In someembodiments, this indicator can identify the tag portion of the one orseveral artifacts and/or portfolios and can, for example, identify thebeginning and/or end of the tagged portion of the one or severalartifacts and/or portfolios. In some embodiments, this indicator can bestored in one of the databases 104 including, for example, the tagdatabase 104-C.

After the indicator of the portion of the one or several artifactsand/or portfolios is received, the process 700 proceeds to block 712wherein an indicator of a criteria is received. In some embodiments, forexample, the indicator of the criteria can be received via one of theuser devices 106 including, for example, the evaluator device 106-A. Insome embodiments, the indicator of the criteria can identify a portionof the evaluation criteria relevant to the indicated portion of the oneor several artifacts and/or portfolios. In some embodiments, thisindicator can be stored in one of the databases 104 including, forexample, the tag database 104-C.

After the indicator of the criteria is received, the process 700proceeds to block 714 wherein an indicator of the trainee is received.In some embodiments, for example, this indicator can be received in oneof the user devices 106 including, for example, the evaluator device106-A. In some embodiments, this indicator can be used to determine theskill level of the trainee in evaluating a portfolio. In someembodiments, this indicator can be stored in one of the databases 104including, for example, the tag database 104-C.

After the indicator of the trainee has been received, the process 700proceeds to block 716 wherein the portfolio evaluation is received. Insome embodiments, for example, the portfolio evaluation can be receivedfrom and/or performed with one of the user devices 106 and/or othercomponents of the evaluation control system 100. In some embodiments,for example, this step can include the grouping of one or several tags,providing the group of one or several tags to the user, and receiving anevaluation based on these tags and the evaluation criteria. In someembodiments, the receipt evaluation can be stored in one of thedatabases 104 such as, for example, the evaluation database 104-D.

After the portfolio evaluation has been received, the process 700proceeds to block 718 wherein the portfolio evaluation and the key arecompared. In some embodiments, for example, this comparison can includecomparing the evaluation of the one or several artifacts and/orportfolios determining if the evaluation and/or score is the same,comparing the tags associated with the evaluation criteria of the firstand second evaluation data to determine similarities and/or differencesin tags applied in both instances, and/or the comparison of portions ofthe one or several artifacts and/or portfolios identified by the tags.In some embodiments, for example, this comparison can be performed bythe processor 102 and/or by the evaluation engine 208 of one of the userdevices 106. In some embodiments, for example, this comparison can beperformed according to a Boolean function wherein matching aspects ofthe portfolio evaluation and the key are assigned a first value andnonmatching aspects of the first and second evaluation data are assigneda second value. In some embodiments, this comparison can be performed inthe same and/or similar manner to the comparison of block 424 of FIG. 4.

After the portfolio evaluation and the key have been compared, theprocess 700 proceeds to decision state 712 wherein it is determined ifthe portfolio evaluation and the key include a different evaluation ofthe portfolio and/or of one or several artifacts in the portfolio. Insome embodiments, for example, this determination can be made by theprocessor 102 and/or by a component of one of the user devices 106including, for example, the evaluation engine 208. In some embodiments,this determination can include retrieving the assigned values indicativeof matching and/or nonmatching aspects of the portfolio evaluationand/or the evaluation of one or several artifacts in the portfolio.

If it is determined that the portfolio evaluation and the key include adifferent evaluation of the portfolio and/or of one or several artifactsin the portfolio, the process 700 proceeds to block 722 wherein anindicator of the difference in the evaluation is stored. In someembodiments, for example, this indicator can be stored in one of thedatabases 104 including, for example, the profile database 104-A, theportfolio database 104-B, and/or the evaluation database 104-D.

After indicator of the difference in the evaluation has been stored, or,returning again to decision state 720 if it is determined that theportfolio evaluation and the key do not include a different evaluationof portfolio and/or one or several aspects in the portfolio, the process700 proceeds to decision state 724 wherein it is determined if differenttags have been assigned to portfolio and/or to the one or severalartifacts in the portfolio in the portfolio evaluation and the key. Insome embodiments, for example, this determination can be made by theprocessor 102 and/or by a component of one of the user devices 106including, for example, the evaluation engine 208. In some embodiments,this determination can include retrieving assigned values indicative ofmatching and/or nonmatching tags in the portfolio evaluation and/or inevaluation of one or several artifacts in the portfolio.

If it is determined the portfolio evaluation and the key includedifferent tags, the process 700 proceeds to block 726 wherein anindicator of the difference in the tags is stored. In some embodiments,for example, this indicator can be stored in one of the databases 104including, for example, the profile database 104-A, the portfoliodatabase 104-B, and/or the evaluation database 104-D.

After indicator of the difference in the tags has been stored, or,returning again to decision state 724 if it is determined that theportfolio evaluation and the key not include the different tags, theprocess 700 proceeds to block 728 wherein a difference report isgenerated. In some embodiments, the difference report can, for example,identify the differences between the portfolio evaluation and the keyincluding, for example, differences in the evaluation of the one orseveral artifacts and/or portfolios, differences in the one or severalartifacts and/or portfolios, and/or differences in the portions of theartifacts and/or portfolios tags identified by the tags. In someembodiments, the difference report can be generated by the evaluationcontrol system 100, and can specifically be generated by the processor102 and/or one of the user devices 106 or component thereof including,for example, the evaluation engine 208.

After the difference report has been generated, the process 700 proceedsblock 730 wherein the difference report is provided. In someembodiments, for example, the difference report can be provided to theuser via one of the user devices including, for example, the supervisordevice 106-B, and specifically the user interface 202 of the user device106.

After the difference report has been provided, the process 700 proceedsto block 730 wherein training is recommended and/or training content isprovided. In some embodiments, for example, the difference between theportfolio evaluation and the key can be sufficient such that additionaltraining can be beneficial. In some embodiments, for example, thistraining can be recommended based on the difference in the evaluation ofthe one or several artifacts and/or portfolios, and in some embodiments,this training can be recommended based on the difference in the tagsapplied to the one or several artifacts and/portfolios. In someembodiments, for example, a component of the evaluation control system100 such as, processor 102 and/or one of the user devices 106 cancompare the difference between one of the portfolio evaluation and thekey with a threshold for requiring additional training, and can, in someembodiments, recommend additional training and/or provide additionaltraining material based of the relationship of the difference inportfolio evaluation and the key.

With reference now to FIG. 8, an exemplary environment with whichembodiments may be implemented is shown with a computer system 800 thatcan be used by a user 804 as all or a component of the evaluationcontrol system 100. The computer system 800 can include a computer 802,keyboard 822, a network router 812, a printer 808, and a monitor 806.The monitor 806, processor 802 and keyboard 822 are part of a computersystem 826, which can be a laptop computer, desktop computer, handheldcomputer, mainframe computer, etc. The monitor 806 can be a CRT, flatscreen, etc.

A user 804 can input commands into the computer 802 using various inputdevices, such as a mouse, keyboard 822, track ball, touch screen, etc.If the computer system 800 comprises a mainframe, a designer 804 canaccess the computer 802 using, for example, a terminal or terminalinterface. Additionally, the computer system 826 may be connected to aprinter 808 and a server 810 using a network router 812, which mayconnect to the Internet 818 or a WAN.

The server 810 may, for example, be used to store additional softwareprograms and data. In one embodiment, software implementing the systemsand methods described herein can be stored on a storage medium in theserver 810. Thus, the software can be run from the storage medium in theserver 810. In another embodiment, software implementing the systems andmethods described herein can be stored on a storage medium in thecomputer 802. Thus, the software can be run from the storage medium inthe computer system 826. Therefore, in this embodiment, the software canbe used whether or not computer 802 is connected to network router 812.Printer 808 may be connected directly to computer 802, in which case,the computer system 826 can print whether or not it is connected tonetwork router 812.

With reference to FIG. 9, an embodiment of a special-purpose computersystem 904 is shown. The above methods may be implemented bycomputer-program products that direct a computer system to perform theactions of the above-described methods and components. Each suchcomputer-program product may comprise sets of instructions (codes)embodied on a computer-readable medium that directs the processor of acomputer system to perform corresponding actions. The instructions maybe configured to run in sequential order, or in parallel (such as underdifferent processing threads), or in a combination thereof. Afterloading the computer-program products on a general purpose computersystem 826, it is transformed into the special-purpose computer system904.

Special-purpose computer system 904 comprises a computer 802, a monitor806 coupled to computer 802, one or more additional user output devices930 (optional) coupled to computer 802, one or more user input devices940 (e.g., keyboard, mouse, track ball, touch screen) coupled tocomputer 802, an optional communications interface 950 coupled tocomputer 802, a computer-program product 905 stored in a tangiblecomputer-readable memory in computer 802. Computer-program product 905directs system 904 to perform the above-described methods. Computer 802may include one or more processors 960 that communicate with a number ofperipheral devices via a bus subsystem 990. These peripheral devices mayinclude user output device(s) 930, user input device(s) 940,communications interface 950, and a storage subsystem, such as randomaccess memory (RAM) 970 and non-volatile storage drive 980 (e.g., diskdrive, optical drive, solid state drive), which are forms of tangiblecomputer-readable memory.

Computer-program product 905 may be stored in non-volatile storage drive980 or another computer-readable medium accessible to computer 802 andloaded into memory 970. Each processor 960 may comprise amicroprocessor, such as a microprocessor from Intel® or Advanced MicroDevices, Inc.®, or the like. To support computer-program product 905,the computer 802 runs an operating system that handles thecommunications of product 905 with the above-noted components, as wellas the communications between the above-noted components in support ofthe computer-program product 905. Exemplary operating systems includeWindows® or the like from Microsoft® Corporation, Solaris® from Oracle®,LINUX, UNIX, and the like.

User input devices 940 include all possible types of devices andmechanisms to input information to computer system 802. These mayinclude a keyboard, a keypad, a mouse, a scanner, a digital drawing pad,a touch screen incorporated into the display, audio input devices suchas voice recognition systems, microphones, and other types of inputdevices. In various embodiments, user input devices 940 are typicallyembodied as a computer mouse, a trackball, a track pad, a joystick,wireless remote, a drawing tablet, a voice command system. User inputdevices 940 typically allow a user to select objects, icons, text andthe like that appear on the monitor 806 via a command such as a click ofa button or the like. User output devices 930 include all possible typesof devices and mechanisms to output information from computer 802. Thesemay include a display (e.g., monitor 806), printers, non-visual displayssuch as audio output devices, etc.

Communications interface 950 provides an interface to othercommunication networks 995 and devices and may serve as an interface toreceive data from and transmit data to other systems, WANs and/or theInternet 818. Embodiments of communications interface 950 typicallyinclude an Ethernet card, a modem (telephone, satellite, cable, ISDN), a(asynchronous) digital subscriber line (DSL) unit, a FireWire®interface, a USB® interface, a wireless network adapter, and the like.For example, communications interface 950 may be coupled to a computernetwork, to a FireWire® bus, or the like. In other embodiments,communications interface 950 may be physically integrated on themotherboard of computer 802, and/or may be a software program, or thelike.

RAM 970 and non-volatile storage drive 980 are examples of tangiblecomputer-readable media configured to store data such ascomputer-program product embodiments of the present invention, includingexecutable computer code, human-readable code, or the like. Other typesof tangible computer-readable media include floppy disks, removable harddisks, optical storage media such as CD-ROMs, DVDs, bar codes,semiconductor memories such as flash memories, read-only-memories(ROMs), battery-backed volatile memories, networked storage devices, andthe like. RAM 970 and non-volatile storage drive 980 may be configuredto store the basic programming and data constructs that provide thefunctionality of various embodiments of the present invention, asdescribed above.

Software instruction sets that provide the functionality of the presentinvention may be stored in RAM 970 and non-volatile storage drive 980.These instruction sets or code may be executed by the processor(s) 960.RAM 970 and non-volatile storage drive 980 may also provide a repositoryto store data and data structures used in accordance with the presentinvention. RAM 970 and non-volatile storage drive 980 may include anumber of memories including a main random access memory (RAM) to storeinstructions and data during program execution and a read-only memory(ROM) in which fixed instructions are stored. RAM 970 and non-volatilestorage drive 980 may include a file storage subsystem providingpersistent (non-volatile) storage of program and/or data files. RAM 970and non-volatile storage drive 980 may also include removable storagesystems, such as removable flash memory.

Bus subsystem 990 provides a mechanism to allow the various componentsand subsystems of computer 802 to communicate with each other asintended. Although bus subsystem 990 is shown schematically as a singlebus, alternative embodiments of the bus subsystem may utilize multiplebusses or communication paths within the computer 802.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A system for verifying the evaluation of subjectmatter, the system comprising: a processor configured to: receive aportfolio, wherein the portfolio comprises a compilation of artifacts,wherein the artifacts comprise work product of an evaluatee; provide oneof the artifacts to an evaluator; receive a plurality of firstindications of an evaluation criteria, wherein the evaluation criteriacomprises a plurality of sub-criteria, wherein the first indications ofthe evaluation criteria identify a first portion of the artifact and afirst portion of the evaluation criteria; assign a value to theplurality of indications of the evaluation criteria, wherein the valueidentifies the source of the indications of the evaluation criteria,wherein the source of the indications of the evaluation criteria is theevaluator; receive a first evaluation of the artifact, wherein theevaluation is based on the evaluation criteria and the first indicationsof the evaluation criteria; receive a second evaluation of the artifact,second indications of the evaluation criteria, and second associatedportions of the artifact; compare the first and second evaluations ofthe artifact, indications of the evaluation criteria, and the associatedportions of the artifact; and provide an indication of the differencesbetween the first and second evaluations of the artifact, indications ofthe evaluation criteria, and the associated portions of the artifact;and memory configured to store: the received portfolio; the firstevaluation of the artifact; the second evaluation of the artifact; thesecond indications of the evaluation criteria and the second associatedportions of the artifact; and the result of the comparison of the firstand second evaluations of the artifact.
 2. The system of claim 1,wherein the processor is further configured to select the evaluator. 3.The system of claim 1, wherein the identified portion of the evaluationcriteria comprises a sub-criterion
 4. The system of claim 1, furthercomprising a user device configured to: display the first evaluation ofthe artifact; display the first indications of the evaluation criteria;and display first associated portions of the artifact.
 5. The system ofclaim 4, wherein the comparison of the first and second evaluations ofthe artifact further comprises: generation of a difference value,wherein the difference value characterizes the degree of differencebetween the first and second evaluations; retrieval of an acceptancethreshold, wherein the acceptance threshold is a value demarking levelsof acceptable and unacceptable differences between the first and secondevaluations; and comparison of the difference value to acceptancethreshold.
 6. The system of claim 5, further comprising comparing thefirst indications of the evaluation criteria and first associatedportions of the artifact with the second indications of the evaluationcriteria and second associated portions of the artifact if thecomparison of the difference value to the acceptance threshold indicatesan unacceptable level of difference between the first and secondevaluations.
 7. The system of claim 5, wherein the processor is furtherconfigured to generate a final score based on the received plurality offirst indications of an evaluation criteria.
 8. A method of verifyingthe evaluation of subject matter comprising: receiving a portfolio,wherein the portfolio comprises a compilation of artifacts, wherein theartifacts comprise work product of an evaluatee; providing one of theartifacts to an evaluator; receiving a plurality of first indications ofan evaluation criteria, wherein the evaluation criteria comprises aplurality of sub-criteria, wherein the first indications of theevaluation criteria identify a first portion of the artifact and a firstportion of the evaluation criteria; assigning a value to the pluralityof indications of the evaluation criteria, wherein the value identifiesthe source of the indications of the evaluation criteria; receiving afirst evaluation of the artifact, wherein the evaluation is based on theevaluation criteria and the first indications of the evaluationcriteria; providing the first evaluation of the artifact, the firstindications of the evaluation criteria, and first associated portions ofthe artifact; receiving a second evaluation of the artifact, secondindications of the evaluation criteria, and second associated portionsof the artifact; comparing the first and second evaluations of theartifact, indications of the evaluation criteria, and the associatedportions of the artifact; and providing an indication of the differencesbetween the first and second evaluations of the artifact, indications ofthe evaluation criteria, and the associated portions of the artifact. 9.The method of claim 8, further comprising selecting the evaluator. 10.The method of claim 8, wherein the identified portion of the evaluationcriteria comprises a sub-criterion.
 11. The method of claim 10, whereinthe comparison of the first and second evaluations of the artifactfurther comprises: generating a difference value, wherein the differencevalue characterizes the degree of difference between the first andsecond evaluations; retrieving an acceptance threshold, wherein theacceptance threshold is a value demarking levels of acceptable andunacceptable differences between the first and second evaluations; andcomparing the difference value to acceptance threshold.
 12. The methodof claim 11, further comprising comparing the first indications of theevaluation criteria and first associated portions of the artifact withthe second indications of the evaluation criteria and second associatedportions of the artifact if the comparison of the difference value tothe acceptance threshold indicates an unacceptable level of differencebetween the first and second evaluations.
 13. The method of claim 10,further comprising generating a final score based on the receivedplurality of first indications of an evaluation criteria.
 14. A methodof training an evaluator comprising: generating a portfolio comprising acompilation of artifacts, wherein the artifacts comprise work product ofan evaluatee; receiving evaluation criteria associated with theportfolio, wherein the evaluation criteria comprise a plurality ofsub-criteria; receiving a key, wherein the key comprises a plurality ofindications of an evaluation criteria, wherein the indications in thekey are the correct indications of an evaluation criteria for aportfolio; wherein the indications of the of the evaluation criteria inthe key identify a portion of one of the artifacts and a sub-criteria ofthe evaluation criteria; providing the portfolio to a trainee; receivinga portfolio evaluation, wherein the portfolio evaluation comprisesplurality of indications of the evaluation criteria, wherein theindications in the portfolio evaluation are received from the trainee;wherein the indications of the evaluation criteria in the portfolioevaluation identify a portion of one of the artifacts and a sub-criteriaof the evaluation criteria; comparing the key and the portfolioevaluation according to a Boolean function to determine the accuracy ofthe portfolio evaluation; and providing an indicator of the accuracy ofthe portfolio evaluation.
 15. The method of claim 14, wherein thecomparison of the key and the portfolio evaluation comprises comparingthe score of the artifact in the portfolio evaluation with the score ofthe artifact in the key.
 16. The method of claim 15, wherein thecomparison of the portfolio evaluation and the key further comprises:generating a difference value, wherein the difference valuecharacterizes the degree of difference between the portfolio evaluationand the key; retrieving an acceptance threshold, wherein the acceptancethreshold is a value demarking levels of acceptable and unacceptabledifferences between the portfolio evaluation and the key; and comparingthe difference value to acceptance threshold.
 17. The method of claim14, wherein the comparison of the key and the portfolio evaluationcomprises the comparison of the indications of the evaluation criteriain the portfolio evaluation to the indications of the evaluationcriteria in the key.
 18. The method of claim 15, further comprisingproviding additional training material.